Bunton\2007.017700\P00-3380 



: Host 'Processor 
lGPfJ|lCPU| [CPUj|CPUl 




SANIC: the System Area Net- 
work interface Con- 
troller FIO interface to 
the host 



indicates a 1 GB +1 GB 
bidirectional FIO link 



to Fibre Channel 
hub and FC-AL 
devices 



EiherNet 



"Graphics 

Figure 1 FIO System Area Network 



WAN and/or LAN con- 
nections to other Hosts 
or FIO Routers. 



An FIO System Area Network 
consists of host processors con- 
nected to multiple IO adapters 
through an FIO fabric made up of 
cascaded switches and routers, 

IO adapters can range in com- 
plexity from single ASIC FIO at- 
tached SCSI adapters to large 
memory rich RAID boxes that 
rival a host in complexity. 



Processor 1 




Host Processor 2 



Process A 
sends a data 
bufferfsj to Pro- 
cess B. 



the data frame 
does not contain 
the destination 
buffer memory ad- 
dress. 




Process B p re- 
allocates where 
to place the 
data. 



Figure 2 Data Transfer with 
Channel Semantics 



Host Pr ocess or 1 



Host Processor 2 




Process A 
sends a data 
buffers) to Pro- 
cess B; 



The data frame 
contains the 
destination 
buffer memory 
address. "". 




Process B previ- 
ously granted per- 
mission for A to 
access its 
memory. 

Figure S Data TVansfer with 
Memory Semantics 
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Processor 1 



Physical Link 
to FIO Fabric 



Figure 4 FIO Client Processes 

Communicates With FIO 
Hardware Through Queue Pairs 




Figure 5 Connected Queue Pairs 



Processor 1 



^tocessor 2 




Figure 6 Connectionless Queue Pairs 




Figure 7 Multiple SANICs per host and multiple ports per SANIC 
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Single Host Name^per 
logical host node 



Single GUID per SANIC 



One or more UDs per 



One or more IPv6 
addresses per SANIC 



Single Switch name per 
switch or packaged 
group of switches 

Single GUID per switch 



One or more UDs per 
switch 

One or more IPv6 
addresses per switch 



Subnet A - > - - - « - 

. Processor 



SANIC*s UDs are unique 
/ within a subnet 



Routers provide connec- 
tions among subnets 




Afl IPv6 addresses 
within a subnel share a 
common set of high 
order address bits. 



Figure 8 Identifying Names for LLEs, 
SANICs,etc 



Figure 9 Subnets and Local Identifiers (LIDs) 




Figure 10 Paths Within and Among Subnets 
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CPU CPU CPU 



|SANiCHMemj ; ~3=fpBT 





Using the message shown in Figure 1 1 on page 37, the Send request mes- 
sage is sent as two frames. Each request frame is in turn broken down into 
4 flits. These ladder diagrams show the request and ACK frames going be- 
tween the source and destination endnodes as well as the request and 
ACK flits between the source and destination of each link. 
This diagram shows a message being sent with a reliable transport. Each 
request frame is individually acknowledged by the destination endnode. 
The second part of the diagram shows the flits associated with the request 
and acknowledgment frames passing among the processor endnodes and 
the two FIO switches. An ACK frame fits inside one flit. One acknowledg- 
ment flit acknowledges several flits. 




SANIC in 
Host Pro- 
cessor A 



SANIC in 
Host Pro- 
cessor B 




SANIC in 
Host Pro- 
cessor A 



SANIC in 
Host Pro- 
cessor B 



Request Frame 
ACK Frame 



Request Flit 
ACK Flit 



Figure 12 Multiple Request Frames (and Flits) and Their 
Acknowledgment Frames (and Flits) 



Figure 13 Single Board Computer 



Self discovering, 
autonomous switch 



Future I/O Link 
(1+1 G Byte/3) 



External Connectors 
•Remote I/O 
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Figure 14 Remote I/O - Active Backplane 




Active Backplane 



Figure 15 Remote I/O - Passive Backplane 




Passive Backplane 
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Figure 16 Chassis-to-Chassis Topology 



Switch ' 



Parallel Fiber 




Copper Cable 



Defined! 
by FIO 1 




Intermediate Fabric 
Element, e.g. a y 
Switch 



Endnode,i.e. a 
Host or IO, 
adapter 




Tfan 



Figure 17 FIO Architecture Layers 



Verbs with parameters to do 
the Send, Receive, RDMA, 
bifid/unbind memory etc. 
functions. 




A welt defined register set to 
allow common management 
code to enumerate the SAN 
IC. This is used for basic dis- 
covery at boot time & at hot . 
plug insertion time. This 
interface is used to know 
which vendor specific driver 
code to load. 



Vendor Specific 
SANIC Driver 





si 


m 


§•8 


CO* 4D 





USER Mode 



KERNEL Mode 



RO Specifies the exact Formats 
and Protocols for 

* Send & RDMA operations. 

* Error Recovery Protocol. 

* Management Functions 

* SAN Path Management 

* Create/Destroy Vt 




Vendor Specific. Not 
specified by RO. 

FIO Specified 
Register Definition 

RO Defined Verbs & 
Parameters 

FIO Defined 
Hardware Interface 



FIO 



Figure 18 FIO SAN, Software and Hardware Layers 



wmemmm. 



Aside from the Formats and ; 
Protocols on the wire, FIO 
does not specify SW internal 
to the adapter. 

Adapter 
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Host SANIC Endnode 



Upper Layer Protocol 



Verbs: 



Transport Layer 



Messages 



Network Layer 



-Frame 



Link Layer 



_BiL 



Physical Layer 



Bits 

T. 



Switch or Router 



Network Layer 



Frame 



Link Layer 



-FflL 



Physical Layer 



Bits 



Link - on-card, copper 
cable, or optical cable 



I/O Adapter Endnode 



Upper Layer Protocol 



Verbs 



Transport Layer 



Messages 



Network Layer 



Frame 



Link Layer 



Flit 



Physical Layer 



Bits 



Fast Layer Description 

Upper Layer Protocol: Ap- 
plication or process which 
employs FlO for communi- 
cating between endnodes. 
Transport: End-to-End mes- 
sage movement, providing 
four types of transport ser- 
vices. 

Network: Frame routing 
through a subnet or multiple 
subnets to its destination 
Urik: Flow-controlled, error- 
controlled, and prioritized 
frame delivery across links. 
Physical: Technology-de- 
, pendent bit transmission and 
reassembly into flits. 



Links - on-card, copper 
cable, or optical cable 



Figure 19 Future I/O Layered Architecture 



Figure 20 Data flow of flit delimiters and flit body data in flit layer 

Byte lanes 





#0 


#1 


#2 


#3 


#4 


#5 


#6 


#7 


#8: 






















Flit Delimiter DW 












mi'- -\^A 


1191 






Flit Body DWO 










V 1 • 








Variable size, between 16B and 512B for data 
: frames. Special delimiter-onty flits with no : 
: body bytes am also, used for link control. 


Flit Body DW N-1 








: l l 


Flit Delimiter DW 








■I 






am 




11111 


Hit Body DWO 
















1 




Variable size, between 1 6B and 51 2B for data 
: . : frames. Special delimiter-only flits with no . • : 
: . body bytes are also used for link control. : " . 


Flit Body DW N-1 


i 
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Figure 21 Flit Delimiter Fields 



Byte 


15 


14 


13 


12 


11 


10 


9 


8 


7 


6 


5 


4 


3 


2 


i 


0 


0 


TYPE 


VL 


SIZE 


rsvd 


2 


SEQ 


4 


FECN 


FFC 


CRVL 


CR 


6 


LCRC 



Flit Header Field Definitions 





Flit Header 
for 

following flit 


TYPE (3b) 


Type of following flit 












VL - Virtual Lane (4b) 


Virtual Lane, in range 0*15 


; i J 




SIZE (6b) 


Flit size = 8 * SIZE (in bytes), 

SIZE = ObOOOOO may either mean 0 bytes, or 512 bytes, depending on 
TYPE - see Figure 22 on page 55. 






SEQ - Sequence Number (16b) 


Flit Sequence Number of following flit, only used for sequenced flits 


: : \ 
ss : 


Network 


FECN (2b) 


Forward Explicit Congestion Notification - End-to- End notification to 
frame destinations that the frames experienced congestion in the fabric - 
this information is fed back to the frame source in the ACK, to assist 
source injection rate control for congestion avoidance. 


Q 




FFC (4b) 


Forward Flow Control - Indication, at each switch stage, of how many 
input ports at the preceding switches in the network have data queued 
for the same output port and VL. This information is aggregated through 
the network, so that the switch arbitration engines at following switch 
stages in the network can adjust policies for greater fairness across 
sources. 




Link Flow con- 
trol and error 
control 


CRVL (4b) 


Virtual Lane for which credit is being given in the CR field 




CR - Credit (4b) 


Number of 64-Byte flit buffer slots that have been freed up to accept 
more flit body data. 






LCRC = Link-level CRC (16b) 


Covers flit body of preceding flit and preceding fields of the delimiter. 
Uses x A 1 6 + x A 1 2 + x A 5 + 1 CRC polynomial. 
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Figure 22 Flit TYPE definition 



Tvpc 


1 loo n e% 


Description 


SIZE 


VL 


SEQ 


CRVL/ 
CR 


Flit 
Body 


vj 


1 ink 

Idie/Ack 


Used when there are no 
other flits to send. 
Acknowledges flits 
received in the opposite 
direction across the link 


SIZE=0; 
size is 8 
bytes 


15 


Carries Seq 
number of 
last flit 
received 
correctly 


0/15 at 
sender 
Ignored 
receiver 


none 


1 


Credit-only 


used to carry credit update 
information when there are 
no Frame flits to send. 


Size=0; 
size is 8 
bytes 


15- 

doesn't 
require 
credit 


0x000- 
0x7FE, 
increment- 
ing 


CRVL indi- 
cates 

which VL is 
being 
given 
transmis- 
sion credit 

CR indi- 
cates how 
many 64- 
byte ftit 
buffer cred- 
its are 
being 
given. 


none 


2 


Frame: 
First 


Frame Flits 

These flits are used for 
transporting frames 
between FIO components. 


Normal 

Flit con- 
tains 
16*SIZE 
oyies oi 
body data. 
(ObOOOOO 
= 512B) 


0-14 for 

Data 

Fames, 

15 for 

Manage- 

ment/Ne 

twork 

Control 

Frames 


0x000- 
0x7FE, 
increment- 
ing 


16B- 
51 2B, 
indicated 
by SIZE 
field 


3 


Frame: 
Middle 


4 


Frame: 
Last 


5 


Frame: 
First and 
Last 


6 

VL is 
used as 
sub-type, 
and no 
credit is 
required 
to send. 


Init 


(Initializes all flit-level 
parameters: VL credit to 0 
on all VLs, clear all ECRC 
accumulators, clear SEQ 
to 0x000, etc.) 


Size=0; 
size is 8 
bytes 


0 


0x7FF 

unse- 

quenced 




none 


Pong - Link 
Control 


Sent to acknowledge 
reception of a Ping flit. 
Not retransmitted. 


Size=0; 
size is 8 
bytes 


r 


0x7FF 

unse- 

quenced 


0/15 at 
sender 
Ignored by 
Receiver 


none 


Ping - Link 
Control 


Used during link initializa- 
tion with the Pong flit to 
time the length of the link 


Size=0; 
size is 8 
bytes 


2* 


0x7FF 

unse- 

quenced 


none 


TOD Con- 
trol Frame 


Time-of-Day frame - 
Not retransmitted, since a 
retransmit would contain 
the *old* (incorrect) time. 


Normal 


3 


0x7FF 

unse- 

quenced 


16B- 
51 2B. 
indicated 
by SIZE 


reserved - 


Ignored by receiver, logged and reported 
as an error 


4-15 




7 


reserved - 


Ignored by receiver, logged and reported as an error 
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Link Flit Logic: 

Building flits from frames, 
link-level flow control and 
error control, etc. 

Link Encoding/Decoding 
Logic: 

link training, byte, word and 
flit alignment, 

encoding and decoding, fre- 
quency difference compen- 
sation. 

Link Physical Layer: 

Digital/analog conversion, 
high/low-speed mux/demux, 
inter-line deskew 



Chip 

Transmit/Receive 
Interfaces 




Transmit Word Stream: 
eight byte words at 
1.25 G Bytes/second 
(8 Bytes/6.4 nsec.) 

Transmit Symbol Stream: 
10 parallel 8b/10b symbols at 
1 .25 G Symbols/second 
(10 Symbols/8 nsec.) 



Physical Media: 
Ten encoded bit streams run- 
ning in parallel over copper dif- 
ferential or optical fiber lines at 
1.25Gbit/sec (0.8 ns/bit) 



Receive Symbol Stream: 
§§|y 10 parallel 8b/10b symbols at 
if 1 GSvmbols/second 



1.25 G Symbols/second 
(10 Symbols/8.nsec.) 



Receive Word Stream: 
eight byte words at 
1 25 GBytes/second 
(8 Bytes/6.4 nsec.) 



Figure 24 FIO Data Path and Interfaces for Link and Physical Layers 
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5B/6B coding for Data 
Characters 


Unencoded 
EDCBA 


Current 

RD- 

abcdei 


Current 

RD+ 

abcdei 


DO: 00000- 
D1: 00001 - 
D2: 00010- 
D3: 00011 


100111 
011101 
101101 
110001 


011000 
100010 
010010 
110001 


D4: 00100- 
05: 00101 
D6: 00110 
D7: 00111 


110101 
101001 
011001 
111000 


001010 
101001 
011001 
000111 


D8: 01000- 
D9: 01001 
010:01010 
011:01011 


111001 
100101 
010101 
110100 


000110 
100101 
010101 
110100 


D1 2:01 100 
D13:01101 
D14:01110 
015:01111 - 


001101 
101100 
011100 
010111 


001101 
101100 
011100 
101000- 


D1 6: 10000- 
D1 7: 10001 
D1 8: 10010 
D19:10011 


011011 
100011 
010011 
110010 


100100- 
100011 
010011 
110010 


020:10100 
D21:10101 
D22:10110 
D23:10111 - 


001011 
101010 
011010 
111010 


001011 
101010 
011010 
000101- 


D24:11000 
D25:11001 - 
026:11010- 
027:11011 - 


110011 
100110 
010110 
110110 


001100 
100110- 
010110- 
001001- 


028:11100 
D29:11101 
D30:11110 
D31:11111 


001110 
101110 
011110 
101011 


001110 
010001 
100001 
010100 



3B/4B coding for Data Characters 


Unencoded 


Current 


Current 


HGF 


RD- 


RD+ 




fghj 


fgnj 


-.0: 000 - 


1011 


0100 


-.1:001 


1001 


1001 


-.2:010 


0101 


0101 


-.3: 011 


1100 


0011 


-.4: 100- 


1101 


0010 


-.5: 101 


1010 


1010 


-.6: 110 


0110 


0110 


-.7: 111 - 


1110/0111 


0001 




5B/6B coding for Special Characters 


Unencoded 


Current 


Current 


EDCBA 


RD- 


RD + 




abcdei 


abcdei 


K28:11100- 


001111 


110000 


K23:10111 - 


111010 


000101 


K27:11011 - 


110110 


001001 


K29:11101 - 


101110 


010001 


K30:11110- 


011110 


100001 



3B/4B coding for Special Characters 


Unencoded 


Current 


Current 


HGF 


RD- 


RD + 






fgnj 


-.0: 000 - 


1011 


0100 


-.1:001 


0110 


1001 


-.2: 010 


1010 


0101 


-.3: 011 


1100 


0011 


-A: 100- 


1101 


0010 


-.5: 101 


0101 


1010 


-.6: 110 


1001 


0110 


-.7: 111 - 


0111 


1000 



Valid Special Characters 



Char Current RD- /Current RD+ 
Name (#) abcdei fghj-abcdei fghj 



K28.0(1C) 
K28.1(3C) 
K28.2(5C) 
K28.3(7C) 
K28.4(9C) 
K28.5(BC) 
K28.6(DC) 
K28.7(FC) 
K23.7(F7) 
K27.7(FB) 
K29.7(FD) 
K30.7(FE) 



001111 0100- 
001111 10 01- 
001111 0101- 
001111 0011- 
001111 0010- 
001111 10 10- 
001111 0110- 
001111 10 00- 
111010 1000- 
110110 1000- 
101110 1000- 
011110 1000- 



110000 1011 
110000 01 10 
110000 1010 
110000 1100 
110000 1101 
- 110000 01 01 
•110000 1001 
110000 01 11 
000101 0111 
•001001 0111 
-010001 0111 
100001 0111 



The comma series b'0011 1110' or 
b'1100 000 V can be used for synchro- 
nization, since it contains a run length 
of 5, which can not appear in any data 
character or combination of data char- 
acters. 



Figu 



re 25 8b/10b Coding Conversion 
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Aux Power Unavailable 



BS1 or BS2 



BS1 



Assert FMsxInt 



Chassis enables Main Power 



Drivers disablt 



Board Primary Power Good 



Anx Power Available 



Board Power-on Init 
(BIST, NVMem Load, etc) . 



B51 




Primary Power Enabled 
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Figure 27 FIO Link Training - One End Power-on shows the typical order of transmission of beaconing 
and link training sequences used in bringing a link from a powered-off state to an alive and operational 
state. 




Link Trained & On Line 



Figure 27 FIO Link Training - One End Power-on 
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Figure 28 Future I/O Layered Architecture 




Intermediate Fabric Element, e.g. a Switch 

V 



mmmmmmi 




Figure 29 Sample Point-to-point Topologies 





Topology one illustrates a SANIC attached di- 
rectly to an endnode either through a cable or 
an ASIC-to-ASIC implementation. 



Topology two illustrates a point-to-point dai- 
sy chain configuration. In this topology, The 
SANIC may only send frames to the directly 
attached endnode. The endnode is responsi- 
ble for forwarding all frames to the subse- 
quent endnode. From the SANIC's 
perspective, all acknowledgments, manage- 



T^S^^ ment, etc. are handled by endnode 1 . 
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Figure 30 Single-board Platform 



Standard Connector 




set to second 
brics. Adapt- 
e responsible 
routing on 
fabric in- 
rs. Future I/O 



Figure 31 Passive Backplane Platform 




Figure 32 Platform-to-Chassis Topology 
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Figure 33 endnodes Connected via External Switch Elements 



FIO 




Chassis are connected via a set of 
switches which are fully meshed 
complete connectivity. 



Each chassis of endnodes contains a 
set of switches which interconnect 
the stations within the chassis. Multi- 
ple switches are provided for avail- 
ability and bandwidth. 



Figure 34 Leaf End-station with Embedded Switch 



IHV ASIC w/embedded 
Switch and SPEs 




Leaf end-station with 
embedded switch 



Figure 35 >: Sample;'Switcli^rame Routing 



DUD = F(route header) 



A frame route header contains a DLID (local desti- 
nation identifier); which is used as a direct index 
into the route, table. The route table may. be [ imple- 
mented in a variety of ways, e.g. a bit mask (one bit 
per port) which indicates which ports a frame tar- 
geting this DLID should be routed. 




Switch Ports 
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Figure 36 Sample Router Route Operation 




Figure 38 Graphical View of Route Headers and Payloads 




Figure 39 Sample Unreliable Multicast Send Operation 



Endnode creates a frame 
which sets the DLED to be 
a previously configured 
multicast DLID. A single 
frame is sent from the 
SAMC to the switch. 



The switch receives the frame, performs a route ta- 
ble look-up and discovers the destination targets 
three output ports. In this example, the switch rep- 
licates the frame (replicates flits as they are re- 
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Figure 37 Attribute Comparison of Switch Routing Technologies 



Attribute 


Wormhole 


Cut-through 


Store-Forward 


Fabric Efficiency - 
Small radius configura- 
tions have essentially 
equivalent efficiency 
regardless of the algo- 
rithm used. 


Medium to large radius fab- 
rics have maximum 60% 
efficiency (random packet 
distribution). Typically, 40- 
50% is reality. 


Due to additional memory 
to deal with head-of-line 
blocking, this algorithm pro- 
vides better efficiency than 
wormhole. 


Depending upon the amount of 
memory within the switch and 
the traffic patterns, this algorithm 
provides better efficiency than 
cut-through or wormhole. 


Design Simplicity 


Simplest design - does not 
require routing table - 
oepenas upon wneiner one 
is using source or destina- 
tion routing. 


Additional complexity due 
to additional packet buffer- 
mg. Kossiuie to implement 
some QoS within the switch 
should congestion occur 
else operation is essen- 
tially wormhole. 


Most complex of the three but 
the impact varies depending the 
amount or uoo, memory compo- 
nent integration, etc. 


Adaptive Routing 


Partial adaptivity (based on 
redundant cables) 


Partial adaptivity - similar to 
wormhole 


Yes. Implementation trade-off in 
terms of management, ordering 
requirements (e.g. deals with 
ghost I/O), etc. 


Deadlock Avoidance - 
no loops within the fab- 
ric allowed. 


Yes, under assumption end 
node makes forward 
progress. 


Similar to wormhole. 


Typically use 802.1 D/OSPF to 
avoid loops. Layer 3 switches 
may also modify packet header 
hop/TTL. 


Cost, i.e. price/perfor- 
mance 


Lowest 


Ltinknr limit lAf/M-mK/\ln U., 

riigner man wormnoie out 
less than store-forward 


Highest relative cost of the three 
due to increased resource man- 
agement. 


Memory Reqs - Each 
requires minimally 2X 
RTT slack buffers 
which may be either 
implemented in on-chip 
or off-chip memory 


Low - Slack buffers and 
optional route table may be 
implemented on-chip 


Typically additional off-chip 
memory for congestion 
buffers 


Off-chip memory - amount varies 
with bandwidth, number of ports, 
possibly fabric radius. 


Flow-control Reqs 


Symbols (dominant) / Credit 


oymDols/Oredits 


oymDols/oredit/None/etc. 


Misc. Resource Reqs 


Minimal management. 
Depends upon whether 
source or destination rout- 
ing. Destination reqs exter- 
nal service frames to 
program route table. 


Similar to wormhole. 


Depends upon what value-added 
capabilities are provided. For 
example, if a layer 3 or layer 4 
switch, then additional manage- 
ment tools/resources required to 
provide policies. 


QoS Capabilities 


Requires VC approach 
which is a limited resource 
and hence limited fabric 
QoS capabilities 


Similar to wormhole. 


Complex QoS capabilities. May 
use variety of tag or flow 
schemes to determine priority of 
service and guarantee band- 
width 


Latency 


Best 


Same as wormhole except 
when congestion occurs 


Complexity of providing store- 
forward buffer management, 
QoS, etc. increases latency com- 
pared to /wormhole 


Fragmentation/Re- 
assembly 


Unlimited frame size but 
often implemented with 
max MTU. FIO specifies 
MTUs for flits and frames. 


Due to limited congestion 
buffering, requires max 
MTU. FIO specifies MTUs 
for flits and frames. | 


Max MTU to avoid skewing 
traffic. FIO specifies MTUs for 
flits and frames. 
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o 



This figure shows two views of connected, reliable transfer 
QPs. Process A on Processor 1 communicates with three pro- 
cesses: processes C and D on Processor 2 and process E on 
processor 3. 

The upper view shows how software might view the connection. 
Buffers in the Send Q flow into buffers in the receive queue on 
the connected QR 

The lower view gives a hardware centric view, showing some of 
the state maintained per connected QP. 



Processor ^ 

lllgcvjjBtM: 



Processor 1 
Process A 

*) ■ . . : ..Qusye Pair 4 
.* OceuePsir.T-; 



Processor 2 




*• Rev Se^ence\# ; = 95 * 



(ID = 27) 



;- j X^ipS^u^cy^g32; 



Figure 41 Connected QPs for Reliable Transfer Operation 
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Processor 1 
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Two views of connectionless. Reliable Datagram 
service. The upper figure shows a software view of 
Reliable Datagram communication among 4 pro- 
cesses on 3 processors. In this example, there is no 
communication among process E and processes C 
and D, otherwise, each QP can send to and receive 
from all the other queue pairs. 



The lower figure shows the multiple queue pairs used by the hardware to 
synthesize the Reliable Datagram service. See section Section 6.7.1 on 
page 167 for more explanation of how this works. \ 
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Figure 42 Connectionless QPs for Reliable Datagram 



Bunton\2007.01 7700\P00-3380 



Processor 1 



m 



This figure shows two views of connection- 
less, unReliable Datagram QPs. Process A 
on Processor 1 communicates with three pro- 
cesses: processes C and D on Processor 2 
and process E on processor 3. 



The upper view shows how software might 
view the connection. Buffers in the Send Q flow into buffers in 
the receive queue on the connected QP. 

The lower view gives a hardware centric view, showing the 
state maintained per connected QP. 
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Figure 43 Connectionless QPs for 
UnReliable Datagram Operation 



Bunton/2007.01 7700/P00-3380 

• # 



Figure 45 Completion queue model 
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Figure 46 Connection establishment accept 
frame transmission sequence 
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Figure 47 Connection establishment reject frame 
transmission sequence 
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Figure 49 QP Attribute Update frame 
transmission sequence 
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Figure 50 - Port States in Spanning Tree 
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Figure 51 - Path Costs to Root (seen by Switch E) 
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Figure 52 - Bundles 
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Figure 53 Node Configuration and Discovery 
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Figure 54 Boot Process 
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Figure 63 Simple Tree with Mixed Bandwidth Links and Adapter Leaves 
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Figure 65 - Simple Tree with Mixed Bandwidth Links and Adapter and Router Leaves 
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